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ABSTRACT:  The  Navy  Probability  of  Raid  Annihilation  (Pra)  Testbed  implements  HLA  federated  simulations  of  ship 
combat  system  elements  against  independent,  reactive  threat  raids  in  a  common  environment  to  formulate  an  overall 
combat  system  assessment.  The  Pra  Testbed  is  a  cornerstone  of  the  U.S.  Navy  Ship  Self  Defense  Test  &  Evaluation 
(T&E)  Enterprise.  The  LPD  17  ship  class  is  the  first  to  implement  a  PRA  Testbed  baseline  as  a  formal  component  of  ship 
class  OT&E.  Products  and  lessons  from  the  LPD  17  baseline  are  being  transitioned  to  multiple  ship  classes  including 
DDG  1000,  LHA  6,  LCS,  and  CVN21. 


The  Pra  Testbed  LPD  1 7  baseline  is  deployed  as  a  secure,  geographically  distributed,  time  managed  federation  with 
three  nodes:  the  U.S.  Naval  Research  Laboratory  (NRL)  Washington,  DC;  Johns  Hopkins  University  (JHU)  Applied 
Physics  Laboratory  (APL)  Laurel,  MD;  and  Naval  Air  Warfare  Center  (NAWC)  Weapons  Division  China  Lake. 
Federation  execution  is  controlled  centrally  from  the  NRL  node. 

The  Pra  Testbed  LPD  17  baseline  federation  is  required  to  conduct  nearly  2000  simulation  test  events  during  the  course 
of  LPD  1 7  Pra  Assessment.  Each  scenario  execution  involves  computation  intensive  calculations  that  progress  much 
slower  than  real  time.  The  development  team  is  using  Run  Automation  software,  dubbed  “OneButtonStart”  to  automate 
batch  runs  of  the  federation  from  a  single  control  node.  OneButtonStart  is  an  important  part  of  the  LPD  17  testing 
program  because  it  dramatically  reduces  manning  requirements  and  takes  advantage  of  all  available  computer  time  to 
minimize  schedule.  Yet,  there  are  numerous  challenges  to  implementing  automated  control  of  a  classified,  non-realtime 
federation  that  includes  both  legacy  and  new  software. 

In  this  paper,  we  describe  the  Run  Automation  capability  and  how  it  will  be  used  in  the  PRA  Testbed  LPD  17  baseline. 
We  also  share  lessons  learned  in  automating  runs  for  a  secure,  geographically  distributed,  time-managed  simulation 
with  multiple  nodes. 


1.  Why  use  a  Run  Automation  Capability? 

The  Navy  Probability  of  Raid  Annihilation  (Pra)  Testbed 
implements  HLA  federated  simulations  of  ship  combat 
system  elements  against  independent,  reactive  anti-ship 
cruise  missile  threat  raids  in  a  common  environment  to 


formulate  an  overall  combat  system  assessment.  The  U.S. 
Navy  Ship  Self  Defense  Test  &  Evaluation  (T&E) 
Enterprise  has  codified  the  PRA  Testbed  as  a  formal 
component  of  ship  class  Operational  Test  &  Evaluation 
(OT&E).  Distributed  simulation  is  being  used  to  satisfy 
OT&E  requirements  that  are  impractical  or  unsafe  to 
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address  with  live  T&E.  The  PRA  Testbed  has  been 
instantiated  in  an  HLA  federation  architecture  which  is: 

•  Time  managed; 

•  Reliant  on  computation  intensive  physics-based 
models; 

•  Geographically  distributed; 

•  Security  classified  at  a  minimum  Secret  level. 

Accreditation  for  OT&E  is  a  tall  order  for  any  simulation 
in  a  resource  constrained  environment.  However,  there 
are  also  practical  concerns  with  achieving  a  successful 
simulation-based  OT&E  event  with  the  PRA  Testbed.  Due 
to  the  complexity  of  the  models,  the  results  for  each  test 
scenario  require  hours  to  compute.  However,  to  assess 
system-of-systems  performance,  a  span  of  thousands  of 
scenarios  is  required.  To  meet  the  OT&E  schedule,  PRA 
Testbed  execution  depends  on  maximizing  use  of  CPU 
time  24  hours  a  day.  Human  intervention  to  initialize  and 
configure  the  Testbed  on  a  per-run  basis  is  impractical; 
hence,  the  need  for  a  run  automation  capability. 

1.1  Federation  Requirements 

The  Navy  PRA  Testbed  is  a  distributed,  classified,  non¬ 
realtime  federation  that  includes  both  legacy  and  new 
software.  The  run  automation  capability  responds  to 
simulation  requirements  driven  by  an  amalgam  of 
technical,  cost,  and  schedule  factors. 

PRA  is  a  platform-level  metric  that  is  levied  on  the 
complete  ship  self  defense  system-of-systems. 
Consequently,  the  simulation  requirements  encompass 
numerous  systems  and  system  interactions,  including 
hardkill,  softkill,  and  ship  signature  control.  Detail  must 
be  sufficient  to  pass  muster  with  the  Navy’s  Operational 
Test  and  Evaluation  Force  (OPTEVFOR)  for  accredited 
use  in  OT&E.  Current  computing  capability  will  only 
support  much  slower-than-real  time  execution  for  many 
of  the  participating  federates  in  the  PRA  Testbed. 
Additional  runtime  performance  constraints  are  levied  by 
the  secure  wide  area  network,  those  these  are  far 
outweighed  by  the  computation  time  necessary  for  the 
complex  mathematical  algorithms  within  each  federate. 

Each  federation  run  provides  the  results  for  a  single  ship 
defense  engagement.  Currently,  each  engagement  can 
take  up  to  two  hours  to  complete.  Adding  runs  increases 
the  overall  time  to  conduct  the  runs  by  days  and  weeks 
rather  than  minutes  and  hours.  To  complete  a  PRA 
assessment  over  the  required  span  of  thousands  of  runs  in 
a  reasonable  amount  of  calendar  time,  it  is  necessary  to 
conduct  runs  24  hours  a  day,  seven  days  a  week.  As  such, 
each  participating  federate  is  required  to  support 
operation  without  human  intervention  and  accept 
commands  from  a  single  control  node. 


Yet,  the  federation  is  also  required  to  run  at  a  classified 
security  level  and  geographically  distributed  across  the 
United  States.  Meeting  those  requirements  while  also 
supporting  automated  24/7  execution  is  challenging.  It  is 
difficult  to  run  a  classified  federation;  adding  the 
complexity  of  multiple  distributed  nodes  in  two  different 
time  zones  makes  implementing  the  simulation  even 
harder.  The  working  hours  of  the  facilities  must  to  be 
correlated  during  testing  to  take  advantage  of  all  model 
experts  for  analysis.  When  one  facility  is  down,  testing 
ceases  until  all  models  can  participate.  Establishing 
graceful  degradation  and  recovery  processes  become 
more  complicated  when  relying  on  automated  control. 
Further,  identification  of  key  or  anomalous  events,  with 
additional  data  collection,  becomes  more  difficult  without 
human  observers  present.  Also,  access  and  control  of 
individual  simulation  federates  through  a  secure  network 
presents  its  own  challenge. 

Lastly,  a  major  hurdle  to  clear  is  successfully  mixing 
legacy  and  new  software  in  the  federation.  While  some 
functionalities  have  been  around  for  years,  there  are  those 
which  have  not  been  modeled  until  now.  The  earlier 
models  were  created  before  HLA  frameworks  were 
standardized  for  government  usage,  and  if  they  have  been 
converted  to  an  HLA  framework  it  was  for  a  specific 
application  and  they  need  to  be  tailored  to  the  Testbed 
architecture.  The  Testbed  also  utilizes  some  re-hosted 
tactical  software.  The  tactical  software  components  have 
particular  expectations  for  operator  interaction,  e.g.,  for 
system  configuration  such  as  mode  settings;  these  must  be 
addressed  to  provide  an  automated  run  capability  for  the 
PRA  Testbed. 

2.  First  Instantiation  of  the  Navy  Pra 
Testbed  for  OT&E 

The  LPD  17  amphibious  ship  class  is  the  first  to 
implement  a  PRA  Testbed  baseline  as  a  formal  component 
of  ship  class  OT&E.  Products  and  lessons  from  the 
LPD  17  baseline  are  being  transitioned  to  multiple  ship 
classes  including  DDG  1000,  LHA  6,  LCS  and  CVN  21. 

The  PRA  Testbed  LPD  17  baseline  is  deployed  as  a  secure, 
geographically  distributed,  time  managed  federation  with 
three  nodes:  the  U.S.  Naval  Research  Laboratory  (NRL) 
Washington,  DC;  Johns  Hopkins  University  (JHU) 
Applied  Physics  Laboratory  (APL)  Laurel,  MD;  and 
Naval  Air  Warfare  Center  (NAWC)  Weapons  Division 
China  Lake.  Federation  execution  is  controlled  centrally 
from  the  NRL  node. 

The  PRA  Testbed  implements  a  “Virtual  Range”  capability 
including  a  common  environment  and  a  defined  threat  set. 
The  LPD  17  baseline  includes  20  different  Scenario 


Natural  Environment  (SNE)  conditions;  the  SNE 
conditions  are  identified  by  combinations  of  two  locations 
(open  ocean,  littoral),  two  seasons  (Summer,  Winter)  and 
five  times  of  day.  Weather  is  limited  to  a  moderate  sea 
state  and  no  precipitation.  Five  different  anti-ship  cruise 
missile  types  are  included  in  the  virtual  range  for  the 
LPD  17  ship  class  assessment.  A  total  of  2000  cases  are 
required  for  LPD  17  assessment. 

3.  Design  of  the  Run  Automation  Capability 

For  the  PRA  Testbed  LPD  17  baseline,  the  Run 
Automation  capability  was  defined  by  the  testing 
constraints  on  the  simulation.  There  were  three  key 
attributes  the  run  automation  capability  must  possess  to 
fully  automate  the  simulation  runs  at  testing: 

1.  ability  to  start-up  and  control  the  status  at  each 
federate’s  state  as  it  transitioned  through  the 
launch,  initialization,  execution  and  termination 
of  each  run; 

2.  ability  to  transition  all  federates  between  the 
termination  of  the  previous  run  to  the  launch  of  a 
new  run  with  changed  variables;  and 

3.  ability  to  monitor  key  events  and  terminate  ‘bad’ 
runs  when  the  federation  fails  to  function 
properly. 

3.1  State  Transition  during  a  Simulation  Run 

The  LPD  17  PRA  Testbed  consists  of  three  nodes  (NRL, 
APL  and  China  Lake)  with  the  federation  management 
controlled  at  NRL.  The  run  automation  capability  will  be 
run  with  the  federation  management  software  at  NRL  to 


control  status  of  the  APL  and  China  Lake  federates.  To 
begin  a  simulation  run,  all  federates  are  to  be  launched 
without  human  intervention.  After  all  federates  have 
completed  start-up,  they  are  initialized  with  a  dedicated 
scenario  including  specific  information  on  the  radials 
relative  to  the  ship,  ship  signature,  threat  type  and  SNE 
information.  The  simulation  run  will  commence  and 
continue  through  the  execution  state.  Once  the  simulation 
run  has  proceeded  through  all  key  events  the  run  is 
terminated  and  all  federates  are  brought  down.  Figure  1 
depicts  the  four  states  of  the  simulation  run  requiring 
transition  management  of  all  federates  by  the  run 
automation  capability. 


Figure  1.  Simulation  Run  States  and  Run  Automation  Process 


3.2  Automated  Transition  from  Run-to-Run 

Also  depicted  in  Figure  1,  is  the  key  attribute  of  the  run 
automation  capability  to  transition  all  federates  between 
termination  of  the  previous  run  to  the  launch  of  a  new  run 
with  changed  variables.  To  perform  2000  simulations 
runs,  the  variables  must  change  slightly  each  run  to  test 
each  separate  case.  For  the  LPD  17  PRA  Testbed,  the  order 
of  simulation  runs  is  listed  in  Table  1  below.  The  order 
will  complete  all  eight  radials  before  changing  the  ship 
signature  and  so  on. 


VARIABLES 

NUMBER 

Radials  (Relative  to  Ship) 

8 

Ship  Signature 

2 

Threat  Number 

5 

Environment  Times  of  Day 

5 

Environment  Seasons 

2 

Environment  Location 

2 

Table  1.  Order  of  Simulation  Runs 


3.3  Early  Termination  of  a  Run 

Time  is  vital  when  managing  the  simulation  runs; 
therefore,  from  a  technical  vantage  point  if  the  federation 
is  not  functioning  properly  it  is  important  not  to  waste 
time  finishing  the  simulation  run.  The  run  automation 
capability  is  required  to  monitor  the  federation  and 
terminate  ‘bad’  runs  when  the  federation  fails  to  function 
properly.  The  term  ‘bad’  run  has  been  described  as  if  a 


federate  failed  to  load  properly,  a  federate  drops  early 
from  the  federation  or  the  correct  scenario  is  not  being  run 
by  all  federates.  However,  it  is  important  not  to  terminate 
‘interesting’  runs  that  contribute  to  the  analysis  required 
to  support  OT&E  requirements.  There  is  a  fine  line 
between  a  ‘bad’  run  and  an  ‘interesting’  run,  but  for  the 
LPD  17  PRA  Testbed  it  has  been  described  as  failure  to 
technically  function  rather  than  failure  in  system 
performance. 


3.4  Key  Events 

Another  mechanism  used  both  to  boost  efficiency  and 
improve  results  analysis  is  monitoring  of  key  scenario 
events.  The  run  automation  capability  tracks  these  events 
to  identify  opportunities  for  early  run  transitions.  If  a 
certain  scenario  event  occurs  during  a  run  (e.g.,  the  radars 
cannot  detect  the  threats  before  a  certain  range,  the  threats 
are  defeated  by  softkill  prior  to  hardkill),  then  it  may  not 
be  necessary  to  complete  the  rest  of  that  run.  Key  events 
are  also  used  to  determine  when  a  run  should  be 
terminated,  (e.g.,  the  last  threat  is  defeated  by  hardkill,  the 
ship  is  hit  by  a  threat).  A  set  of  these  key  scenario  events 
was  determined  federation-wide.  The  federates  can  use 
these  “Key  Events”  to  determine  the  state  of  the 
simulation  run.  All  federates  can  also  use  these  events  to 
assist  in  collecting  detailed  data  for  after  action  review. 
The  Key  Events  are  used  by  the  run-automation  capability 
to  make  run  transition  more  efficient.  Since  there  may  be 
no  human  in  the  loop  during  run-automation  trials 


tracking  these  events  will  allow  the  run  automation  to 
transition  between  runs  more  efficiently  allowing  for 
more  simulation  runs  in  a  given  time  period. 

4.  Implementation 

For  the  LPD  17  PRA  Testbed,  the  run  automation 
capability  is  located  in  the  Scenario  Environment 
Federate  (SEF).  The  SEF  consists  of  three  software 
applications:  OneButtonStart,  Federation  Control  and  Run 
Automation.  The  incremental  software  development  for 
this  capability,  shown  in  Figure  2,  was  progressed  as  the 
separate  applications  were  implemented  and  tested. 
Building  on  the  basic  Federation  Control  functionality 
was  the  OneButtonStart  application  which  automates  the 
start-up  and  termination  of  the  federation.  Once  the 
OneButtonStart  functionality  was  performing,  the  Run 
Automation  application  could  be  fully  integrated  which 
allows  the  transition  between  simulation  runs. 


Run  Automation 

OBS  OBS 

Federation  Control  Federation  Control  Federation  Control 


Figure  2.  Run  Automation  Capability  Software  Buildup 


4.1  OneButtonStart  and  Federation  Control 

Basic  Federation  Control  functionality  includes 
initialization,  identification,  synchronization,  monitoring 
and  termination  of  the  federation.  To  implement  a  run 
automation  capability,  these  Federation  Control  functions 
required  cohesive  management  with  the  automated 
federation  start-up  and  termination  software, 
OneButtonStart.  Both  of  these  functionalities  are  located 
in  the  SEF  federate;  however,  to  participate  in  the 
automated  management  each  federate  must  hold  an 
application  of  the  OBS  software  locally.  To  activate  the 
OBS  software,  the  federation  manager  at  the  control  node 


opens  the  OBS  application  which  in  turn  connects  to  all 
machines  with  an  OBS  application  loaded.  In  this 
instance,  the  OBS  application  is  launching  one  simulation 
run  for  the  federation. 

Focated  below  in  Figure  3  is  the  process  followed  by  SEF 
to  manage  the  OBS  and  Federation  Control  functions. 
Note  that  SEF  is  listening  and  sending  specific  objects 
and  interactions  to  start  up,  manage,  and  terminate  the 
federation.  Note  in  Figure  4,  the  SEF  Graphic  User 
Interface  (GUI),  the  Federation  Status  Table  which 
monitors  current  status  of  all  federates  at  all  times  during 
a  simulation  run. 


Figure  3.  Detailed  OneButtonStart/F ederation  Control  States 


File  Menu 


3  ►File 


Join/Resign  Button 
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SEF  2. 1C  /  20Feb2006:1400 

Configuration:  sef.conf-<- 
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Configuration  File 
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Elapsed  Time:  0: 1:0 
Real  Time/Sim  Time:  0.045011252? 


Figure  4.  SEF  GUI 


4.2  Run  Automation 

While  OBS  and  the  Federation  Control  can  automate  the 
start-up,  initialization,  execution  and  termination  of  the 
federation,  the  Run  Automation  application  allows  the 
federation  to  transition  between  simulation  runs.  Instead 
of  loading  the  OBS  application  with  one  simulation  run,  a 
batch  file  of  simulation  run  parameters  is  loaded.  The  Run 
Automation  application  then  monitors  the  OBS  as  it 
transitioned  to  the  next  simulation  run. 

The  Run  Automation  application  can  be  made  as 
intelligent  as  the  federation  requires.  In  the  LPD  17  Pra 
Testbed,  the  Run  Automation  application  is  being  used  to 
maximize  the  CPU  usage  for  all  24  hours  in  a  day  without 
the  need  for  a  human  in  the  loop.  Therefore,  the  run 
automation  application  monitors  the  Federation  Control 


application  and  terminates  ‘bad’  runs  when  the  federation 
fails  to  function  properly.  The  Run  Automation 
application  will  reload  the  same  simulation  run  to  test  if 
the  error  occurs  again,  but  will  then  move  on  to  the  next 
simulation  run.  After  each  simulation  run,  the 
configuration  file  and  simulation  run  results  are  stored  in 
separate  files. 

5.  Way  Forward 

With  the  PRA  Testbed  LPD  17  baseline  runs-for-score 
imminent  in  the  second  half  of  2007,  there  are  many 
lessons  to  be  learned  in  the  near  future.  In  the  time 
leading  up  to  the  Final  Build  of  the  LPD  17  federation, 
the  Run  Automation  capability  will  continue  to  mature  in 
its  ability  to  intelligently  manage  the  simulation  runs. 


This  capability  has  because  an  integral  part  of  the 
federation  management  as  it  dramatically  reduces 
manning  requirements  and  takes  advantage  of  all 
available  computer  resources. 

Though  computing  power  continues  to  increase,  so  does 
the  complexity  of  the  ship  self  defense  system-of-systems. 
Computation  intensity  required  to  meet  OPTEVFOR 
requirements  for  OT&E  accreditation  will  not  wane.  As 
such,  runtime  efficiency  of  the  PRA  Testbed  will  remain  a 
challenge  and  the  Run  Automation  capability  will  remain 
a  key  part  of  meeting  the  challenge. 
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